Entropy and thinning of discrete random variables

We describe five types of results concerning information and concentration of discrete random variables, and relationships between them. These results are motivated by their counterparts in the continuous case. The results we consider are information theoretic approaches to Poisson approximation, the maximum entropy property of the Poisson distribution, discrete concentration (Poincaré and logarithmic Sobolev) inequalities, monotonicity of entropy and concavity of entropy in the Shepp–Olkin regime. 1 Results in continuous case In this paper we give a personal review of a number of results concerning the entropy and concentration properties of discrete random variables. For simplicity, we only consider independent sets of random variables (though it is an extremely interesting open problem to extend many of the results to the dependent case). These results are generally motivated by their counterparts in the continuous case, which we will briefly review, using notation which holds only for Section 1. For simplicity we restrict our attention in this Section to random variables taking values in R. For any probability density p, write λp = ∫∞ −∞ xp(x)dx for its mean and Varp = ∫∞ −∞ (x − λp)p(x)dx for its variance. We write h(p) for the differential entropy of p, and interchangeably write h(X) for X ∼ p. Similarly we write D(p‖q) or D(X‖Y ) for relative entropy. We write φμ,σ2(x) for the density of Gaussian Zμ,σ2 ∼ N(μ, σ). Given a function f , we wish to measure its concentration properties with respect to probability density p; we write λp(f) = ∫∞ −∞ f(x)p(x)dx for the expectation of f with respect to p, write Varp(f) = ∫∞ −∞ p(x)(f(x)− λp(f))dx for the variance and define